Skip to content

Add concept of block events and some design notes #1059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

Conversation

ml-evs
Copy link
Member

@ml-evs ml-evs commented Mar 4, 2025

This PR adds the concept of "events" that can be registered on block types; essentially, server-side functions that can be triggered separately from parsing/ingestion for things like setting parameters. It also introduces the handling of particular events raised by Bokeh plots in the base data block in Vue.

This is all described in a new design document for this current version of datablocks, also committed in this PR.

I think eventually we can add several lifecycle hooks to the base block for handling certain things (remake block from scratch etc) that can be reused by derived block types, though I haven't put any effort into doing this now. I'm already using this functionality in a custom block that sets some parameters based on user feedback that requires reanalysing all block data (essentially a slider setting some offset time parameter).

For review, I recommend reading the design doc in the rtd preview at https://the-datalab--1059.org.readthedocs.build/en/1059/design/blocks/.

There's still a few open questions about how errors in events should be handled, and what methods should be triggered afterwards (mostly due to the fact we always call the to_web() block method in the update-block route.

Copy link

codecov bot commented Mar 4, 2025

Codecov Report

Attention: Patch coverage is 82.43243% with 13 lines in your changes missing coverage. Please review.

Project coverage is 74.38%. Comparing base (d7f5b80) to head (9ae754c).

Files with missing lines Patch % Lines
pydatalab/src/pydatalab/bokeh_plots.py 30.00% 7 Missing ⚠️
pydatalab/src/pydatalab/routes/v0_1/blocks.py 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1059      +/-   ##
==========================================
+ Coverage   74.28%   74.38%   +0.10%     
==========================================
  Files          66       66              
  Lines        4487     4556      +69     
==========================================
+ Hits         3333     3389      +56     
- Misses       1154     1167      +13     
Files with missing lines Coverage Δ
pydatalab/src/pydatalab/apps/xrd/blocks.py 74.66% <100.00%> (+1.42%) ⬆️
pydatalab/src/pydatalab/blocks/base.py 82.35% <100.00%> (+8.72%) ⬆️
pydatalab/src/pydatalab/routes/v0_1/blocks.py 45.36% <0.00%> (-3.00%) ⬇️
pydatalab/src/pydatalab/bokeh_plots.py 79.66% <30.00%> (-2.16%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link

cypress bot commented Mar 4, 2025

datalab    Run #3535

Run Properties:  status check failed Failed #3535  •  git commit aa6afd8af7 ℹ️: Merge 9ae754c3d7117aa8f271b9e0b1ee6591f8c20e92 into d7f5b8027f62b048bb5e22328acf...
Project datalab
Branch Review ml-evs/blocks
Run status status check failed Failed #3535
Run duration 05m 27s
Commit git commit aa6afd8af7 ℹ️: Merge 9ae754c3d7117aa8f271b9e0b1ee6591f8c20e92 into d7f5b8027f62b048bb5e22328acf...
Committer Matthew Evans
View all properties for this run ↗︎

Test results
Tests that failed  Failures 6
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 144
View all changes introduced in this branch ↗︎

Tests for review

Failed  cypress/e2e/editPage.cy.js • 2 failed tests • End-to-end tests (chrome)

View Output

Test Artifacts
Edit Page > Uploads an XRD file, makes an XRD block and checks that the plot works Test Replay Screenshots
Edit Page > Uploads an Raman data file, makes a Raman block and checks that the plot is shown Test Replay Screenshots
Failed  cypress/e2e/editPage.cy.js • 2 failed tests • End-to-end tests (firefox)

View Output

Test Artifacts
Edit Page > Uploads an XRD file, makes an XRD block and checks that the plot works Screenshots
Edit Page > Uploads an Raman data file, makes a Raman block and checks that the plot is shown Screenshots
Failed  cypress/e2e/editPage.cy.js • 2 failed tests • End-to-end tests (electron)

View Output

Test Artifacts
Edit Page > Uploads an XRD file, makes an XRD block and checks that the plot works Test Replay Screenshots
Edit Page > Uploads an Raman data file, makes a Raman block and checks that the plot is shown Test Replay Screenshots

@ml-evs ml-evs marked this pull request as ready for review March 10, 2025 22:06
@ml-evs ml-evs force-pushed the ml-evs/blocks branch 2 times, most recently from 2f892ca to f765a97 Compare March 19, 2025 18:56
@ml-evs
Copy link
Member Author

ml-evs commented Mar 19, 2025

Just pinging @BenjaminCharmes and @jdbocarsly for review on this one!

Copy link
Contributor

@BenjaminCharmes BenjaminCharmes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀 very promising!
XRD Block works perfectly fine.

@@ -27,9 +27,19 @@ class XRDBlock(DataBlock):
def plot_functions(self):
return (self.generate_xrd_plot,)

@classmethod
@event
def set_wavelength(self, wavelength: float | None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you show how this gets implemented in the XRD block before we merge? That would be a good way to test that everything is working on the front end

@@ -207,6 +207,10 @@ export default {
this.contentMaxHeight = "none";
}
});
document.addEventListener("bokehStateUpdate", this.handleBokehEvent);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this adds a listener to the overall document, will we have multiple listeners if we have multiple blocks on a page? This could cause the handler to be called multiple times

you should be able to add this listener to a only the relevant node by assigning the outer div ref="thisBlock", then only add the listener to:
this.$refs.thisBlock.addEventListener("bokehStateUpdate", this.handleBokehEvent);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still struggling a lot with this... following the approach with $refs, it seems that the block events bubble up to the block-list-item defined in the EditPage and skip the DatablockBase.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps the easiest resolution is adding a ref to the DataBlockBase and then directly calling:

<template>
 <DataBlockBase :item_id="item_id" :block_id="block_id" ref="xrdBlockBase">
</template>
...
this.$refs.xrdBlockBase.handleBokehEvent(event);

@ml-evs
Copy link
Member Author

ml-evs commented Mar 21, 2025

Made some fixes post-discussion with @jdbocarsly. Known limitations still:

  • The event handler currently triggers per block on the page, with the wrong data. We need to find a way to scope the handler, and then dispatch it correctly to the right block. Ideally this would be encapsulated in a function so that we don't have too much fiddly boilerplate in the Bokeh callbacks.
  • We should probably rename bokehStateUpdate
  • The whole block currently updates; it would be nice to be able to also do partial updates

@ml-evs ml-evs force-pushed the ml-evs/blocks branch 2 times, most recently from f7fc272 to 34c71dd Compare April 4, 2025 23:16
@ml-evs ml-evs force-pushed the ml-evs/blocks branch 2 times, most recently from 827928a to 3f6c7e2 Compare April 21, 2025 14:46
@ml-evs ml-evs added this to the v0.6.x milestone May 9, 2025
@ml-evs ml-evs modified the milestones: v0.7.x, v0.6.x May 20, 2025
@ml-evs ml-evs self-assigned this Jun 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants